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LONG-TERM  GOALS 

Our  long-term  goal  is  to  develop  a  modeling  framework  to  predict  sediment  transport,  the  evolution  of 
seafloor  roughness,  and  acoustic  propagation  through  the  seafloor  in  the  nearshore  and  littoral 
battlespace  environment. 

OBJECTIVES 

Our  primary  objective  is  to  implement  a  massively  parallel  version  of  an  existing  mixture  model, 
SedMix3D  (Penko  and  Slinn,  2006),  for  simulating  small-scale  ripple  dynamics  in  shallow  littoral 
environments.  Applicability  of  the  existing  serial  version  of  SedMix3D  is  severely  hampered  by 
physical  limitations  (memory  and  CPU  speed)  of  typical  desktop  workstations.  The  scalable  version  of 
SedMix3D  developed  here  will  be  able  to  simulate  prototype  size  domains  found  in  the  center  of  a 
laboratory  U-tube  (up  to  1  m  in  length).  The  parallel  version  of  SedMix3D  is  a  powerful  research  tool 
that  will  be  used  to  study  the  details  of  small-scale  sand  ripple  dynamics  including  (1)  the  effects  of 
suspended  sediment  concentration  on  turbulence  modulation,  (2)  the  dynamics  of  ripple  transitions 
from  2D  to  3D  (and  back  to  2D)  under  changing  forcing  conditions,  and  (3)  the  role  of  terminations 
and  bifurcations  on  ripple  migration  and  growth  rates. 

APPROACH 

SedMix3D  treats  the  fluid-sediment  mixture  as  a  single  continuum  with  effective  properties  that 
parameterize  the  fluid-sediment  and  sediment-sediment  interactions  using  several  closures  for  the 
sediment  phase.  The  capability  of  SedMix3D  to  simulate  small-scale  sand  ripple  dynamics  has  been 
illustrated  both  qualitatively  and  quantitatively  (Penko  and  Slinn,  2006;  Penko  et  ah,  2008). 
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Unfortunately,  the  most  rigorous  quantitative  comparisons  between  the  model  and  existing  laboratory 
data  have  been  performed  only  using  domains  that  are  essentially  2D.  The  expensive  computational 
costs  coupled  with  hardware  limitations  on  typical  desktop  workstations  have  limited  the  largest  scale 
3D  simulation  performed  with  the  serial  version  of  SedMix3D  to  date  to  a  domain  about  12  cm  in 
length,  which  required  75  days  of  computations  to  simulate  just  about  9s  of  real  time.  Utilizing  HPC 
resources  available  through  an  ongoing  NRL  6. 1  base  project,  we  seek  to  increase  the  domain  size  by 
up  to  an  order  of  magnitude  while  reducing  the  wall-clock  run  times  by  up  to  2  orders  of  magnitude, 
running  on  processor  arrays  from  64  up  to  a  maximum  available  of  2048  cores  on  the  Cray  XT3  at  the 
ERDC  DSRC  and  the  Cray  XT5  at  the  NAVY  DSRC. 

WORK  COMPLETED 

We  have  recently  completed  parallelization  of  the  SedMix3D  code  using  the  message-passing  interface 
(MPI).  Preliminary  simulations  have  been  performed  on  processor  arrays  up  to  5 12  cores  on  the  Cray 
XT3  at  the  ERDC  DSRC.  We  have  achieved  our  goal  of  simulating  prototype  size  domains,  with  the 
largest  simulation  to  date  roughly  covering  a  ~  1  m2  sand  bed  (Figure  1).  We  have  conducted  initial 
performance  scaling  tests  and  rigorous  quantitative  validation  of  the  simulation  output  is  underway. 
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Figure  1.  Initial  sediment  bed  of  parallelized  three-dimensional  SedMix3D  numerical  model. 
The  domain  is  initialized  with  8  ripples  with  1  cm  heights  and  10  cm  wavelengths  on  a  1  cm  packed 
bed.  The  grain  size  diameter  of  the  sediment  is  0.4  mm.  Only  4  cm  of  the  16  cm  domain  height  is 
shown  here.  The  flow  over  the  ripples  can  be  oriented  at  varying  angles  to  observe  three- 

dimensional  ripple  evolution. 
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RESULTS 


We  recently  perfonned  a  fixed  scaling  speedup  test  using  the  simulation  domain  shown  in  Figure  1. 
The  new  parallel  version  of  SedMix3D  was  run  with  arrays  of  64,  128,  256,  and  5 12  cores  on  the  Cray 
XT3  at  the  ERDC  DSRC.  The  scalability  of  the  code  up  to  5 12  cores  is  very  encouraging,  resulting  in 
approximately  3.5  times  speedup  over  64  cores.  We  expect  that  continued  scalability  will  require  more 
sophisticated  I/O  algorithms.  As  the  number  of  cores  increases  it  becomes  exceedingly  difficult  and 
time  consuming  to  write  results  to  disk.  We  will  continue  testing  and  validation  of  the  parallel  version 
of  SedMix3D  with  high-resolution  laboratory  data  in  FY10. 


SedMix  3D  is  parallelized  using  Message-Passing  Interface  (MPI)  programming.  Shown  is  a  plot  of 
the  increase  in  computational  speedup  running  a  fixed  size  problem  on  an  increasing  number  of 
processors.  The  code  runs  approximately  3.5  times  faster  on  512  processors  than  on  64  processors. 

IMPACT/APPLICATIONS 

The  research  tool  developed  here  will  be  used  to  improve  understanding  of  bedform  dynamics,  and 
more  generally,  bottom  boundary  layer  physics  in  shallow  sandy  environments.  We  expect  the  results 
to  impact  broadly  both  Naval  operations  (e.g.,  littoral  navigation  and  trafficability  including  surf  zone 
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breaching,  MCM,  and  characterization  of  both  the  air-sea  boundary  layer  and  the  bottom  boundary 
layer  in  the  battlespace  environment)  and  commercial  activities  (e.g.,  coastal  restoration  procedures 
and  design  of  coastal  infrastructure). 

RELATED  PROJECTS 

The  PI  is  the  lead  investigator  for  an  NRL  6. 1  base  project  (FY08-FY10),  titled  “Direct  Numerical 
Simulation  of  Small-scale  Sand  Ripples”.  The  parallel  version  of  SedMix3D  developed  here  is  being 
used  in  direct  support  of  the  NRL  base  project.  Likewise,  all  HPC  resources  used  to  perform  the  work 
funded  by  ONR  were  provided  by  NRL.  Additionally,  the  model  and  simulation  results  produced  here 
form  the  basis  for  the  Ph.D.  dissertation  (expected  August  2010)  of  student,  Ms.  Allison  M.  Penko, 
University  of  Florida.  Presently,  Ms.  Penko  is  working  at  NRL  under  the  Student  Temporary 
Employment  Program  (STEP). 
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